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AESTRACT 



The least understood of complex systems is that 
which produces educated people* Superficially, many key elements cf a 
manufacturing organization are part of the system: a product, product 
tolerance limits, methods for assessing product deviation, and a 
feed-back system for making compensatory adjustments. In layman*s 
terms these elements are: educated youth, educational goals or 
objectives, performance tests, and a reporting system. Their 
existence has led to an attempt at applying a wholesale systems 
framework tc education with mixed results. Reasons for the lack cf 
success in this attempt is analysed in this study. It is suggested 
that system analysts, rather than modeling the total system, should 
focus cn those aspects of education which can be systematized. Such 
aspects are: resource allocation, scheduling, prediction systems 
(research and evaluation) , and data bank systems. It is theorized 
that two conditions must be met to enable a systems approach to be 
successful in an educational application: (1) The designers must 

exhibit a fine sensitivity to the fact that schools are political 
entities, and (2) the system must be focused at the level of the 
individual student. The implications of these assumptions are 
discussed. (Author) 




TA 00 0 315 , EDO 4572 d 



NEW JERSEY URBAN SCHOOLS DEVELOPMENT COUNCIL 



MONOGRAPH #4.000 



Evaluation Occasional Paper 2 



AN ANALYSIS OF 
SYSTEMS APPLICATIONS 
IN EDUCATION 



U.s, DEPARTMENT OF HEALTH. EDUCATION 
& WELFARE 
OFFICE OF EDUCATION 
THIS DOCUMENT HAS BEEN REPRODUCED 
EXACTLY AS RECEIVED FROMTHE PERSON OR 
ORGANIZATION ORIGINATING IT. POINTS OF 
VIEW OR OPINIONS STATED DO NOT NECES- 
SARILY REPRESENT OFFICIAL OFFICE OF EDU- 
CATION POSITION OR POLICY. 



O 

ERIC 



AN ANALYSIS OF SYSTEMS APPLICATIONS IN EDUCATION 



Daniel Landis 

Educational Testing Service 
Princeton,, New Jersey 



Ernest J. Anastasio 
Educational Testing Service 
Princeton, New Jersey 



Robert M. Slivka 

New Jersey Urban Schools Development Council 
Trenton, New Jersey 



\ 

and 



John L. Hayman, Jr. 
Research Council of the Great Cities 
Washington, D.C. 



O 

ERIC 



An Analysis of Systems Applications in Education^ 

♦ 

The large and some would say vast, recent expenditures for American 
education, e.g., 36 billion dollars in 1964, have led inexorably to demands that educa- 
tors show the value of their activities. The catchword is now "accountability • " 

To underscore the seriousness of the concern and indeed the lack of confidence 
in the current educational enterprise, most program grants have mandated eval- 
uation components. Faced with this situation, most practicing educators, not 
having an internal evaluation capability within their school systems, turned to 
the research community for help. University researchers became involved, and 
in the process discovered that the methodologies at their command were completely 
inadequate. They were involved in a different area, which has come to be termed 
"evaluation," and Egon Guba (1969) expressed the frustrations of researchers 
with his statement that "evaluative inquiry does little more for a researcher 
than to give him an opportunity to disgrace himself while not really serving his 
purposes." 

Faced with a pressing need, educators began searching for a new approach 
that would serve several functions. First of all, the approach would have to 
provide evaluative data on functioning programs. Second, it would have to be 
able to relate such data to cost factors (this function became critical as more 
and more school bond issues were turned down by the taxpayers and state legisla- 
tors seemed to be more interested in their own salaries and expense accounts than 
in education). Third, the approach would have to be capable of making predictions 
on the effectiveness of changing program components. Such functions seemed ready 
made for the group of engineers which Boguslaw (1965) has called the "new Utopians," 
more familiarly, systems scientists. 

There were several other reasons why the systems approach seemed applicable, 
not the least of which was that it was being pushed by the Federal funding agencies. 



Beyond this, however, were such rationales as: 1) the approach used computers, 

and the apparently large storage capacities of such machines seemed to be neces- 
sary to handle the complexities of the educational process; 2) the field had a 
reputation for successfully dealing with enormously complex operational systems, 
e.g., the space effort; 3) finally, many educators became convinced that it was 
valid to see education as a special case of a manufacturing system* 

Thus, it has become somewhat commonplace to gpeak of elements of the edu-r 
cational enterprise in manufacturing terms* For example, educated youth is 
taken to mean the product of an educational system* Similarly, we can view edu- 
cational goals as being a statement of the tolerance limits for the system, per- 
formance tests as the methods for assessing deviation from the limits, and presumably 
report cards and data banks are the ways in which feedback for assessing total system 
performance is accomplished as well as providing stockholder (taxpayer) reports. 

Given the enormous problems of education and a certain superficial similarity 

to a manufacturing system, it was not long before educators were deluged with a new 

{ 

breed of enterpreneur : the systems scientist and the hardware salesman, often if 
not the same person then one so similar that they appeared as Tweedle-dum and Tweedle 
dee. It was no accident that the software of the **new utopian** was dependent upon 
an investment in some rather expensive equipment — often a computer that the school 
system found it didn’t need, couldn’t program, and couldn’t afford. 

The services proposed, and sometimes even provided, by such persons have 
taken one of two basic forms. The first is exemplified by two studies, one done 
for the Kentucky school system (Miller, 1968), and one done on Project Head Start 
(Watts and Horner, 1968). This approach may be fairly termed "old wine in new 
bottles.** The old wine is the familiar multiple regression, and sometimes, multiple 
canonical, methodology; the new bottle is the adding of cost information and the 
calling of the whole pursuit "systems.” 
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The second form is exemplified by the work of Sisson (1967), who is attempt- 
ing no less than a modeling of the entire educational process. In this approach, 
the systems analyst attempts to predict achievement (never mind how it is measured) , 
from a knowledge of some rather simple input parameters (in Sisson's case, staff 
per student, material per student [in dollars], space allocation, community re- 
lations effort [in dollars] and parental education). This approach differs from 
the multiple regression method in that bivariate normality and linearity of re- 
gression are not assumed to be the only relationship holding between variables. 

The writing of functional equations which reflect other kinds of relationships, 
e.g., exponential, is certainly an improvement over the regression approach if 
for no other reason than the fact that real systems are distressingly nonconform- 
ist with regard to Gaussion distributions. Unfortunately, the unreliability 
inherent in most measures of human, let alone system . performance, requires that 
constants be added to the equations in order to bring the plotted functions into 
line with real data. And, as one wag commented with regard to Clark Hull's use 
of constants, "an equation with two constants can be made to describe an elephant, 
add a third and it will wag its tail." 

The modelers, at least those in the Sisson tradition seem also to assume 
that feedback of information on system performance occurs with a zero delay time. 

We say, "assume," since few if any mathematical equations used in these educa- 
tional applications include such functions. However, it takes no great insight 
into the ways in which educational systems function, to note how much inertia 
there is in responding to knowledge that the "system n is not functioning properly. 
Often, by the time action has been taken, the problem has become exacerbated to 
the point that the solution adopted has little (if any) real effect. In modeling 
real educational systems, the feedback system must not only have an extraordinarily 
long time constant, but the loop must include delays and modifications of input 
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arising from the varied responsibilities of decision makers at several levels. 

The problems of modeling such a loop are yet to be understood, in contrast to 
ordinary manufacturing operations where decision making functions are centralized 
and response to a problem can be timely* 

If anything, an educational decision maker needs to develop .a precognitive 
(Krendel and McRuer, 1960) mode of operation* That is, he has to be able to 
anticipate the system's performance and act ahead of the predicted changes* While 
methods for the presentation of feedback leading to such modes of operation have 
been worked out for certain skills (e*g*, flying an airplane or driving a car), 
they have yet to be developed for large and complex systems* Sisson-like models, 
while they may describe, more or less accurately, relationships between input and 
output variables at a given time, become progressively more and more out of phase, 
or inaccurate, as time passes , because they fail to take account of the open- loop- 
like characteristics of the educational system. Until such considerations can be 
built into the model, these approaches will be inaccurate guides to decision making. 

In any case, these two approaches are focused at the administrative level 
and bridging the gap to individual students is extremely difficult. There can be 
little disagreement with Thomas Watson (1969), IBM’s President, in his statement 
that the critical problem of education is how to articulate the functioning of a 
complex system with the needs of the individual student* It is at this level that 
we see a need to focus the major impact of the systems approach, and it is to this 
problem that the remainder of this paper is directed. 

To provide adequate management of an educational system, and thereby increase 
the probability of making intelligent and informed decisions, the educational administ- 
rator or decision maker needs to have the following (Provus, 1969): 

A statement of goals. 

A plan of operation. 

A statement of support systems. 

-4- 
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(1) 

( 2 ) 
(3) 



(4) An evaluation system for monitoring the 

installation and operation, with provision 
for feedback. 

Each of these is critically important for determining the degree of success of a 
total educational program. However, the fourth point (an evaluation monitoring system) 
is of particular importance in the design and construction of a system which is useful 
in predicting individual performance. Obviously^ an evaluation structure that provides 
continuous monitoring and feedback at all levels of the educational process is needed 
at those levels which have heretofore been ignored. 

The key attribute of such a system would seem to be the introduction of an in- 
tegrated set of man/machine programs and functions which can actively monitor re- 
lationships among the three major factors (Miller, 1968) of any educational system 
at both the student and at the administrative level. These major factors are: 

(1) input factors, including community and socio-economic variables that have been 
traditionally incorporated into system evaluations; (2) process factors, including 
a variety of school and instructional variables only a few of which have been analyzed 
and (3) output factors, including a number of student achievement and school achieve- 
ment variables that have generally been overlooked or ignored in system studies, e.g,, 
affective variables. 

These factors are reflected by a series of propositions which grow out of a 
particular view of education and out of the role systems analysts might assume in 
helping educators achieve their goals: 

(1) Education is concerned with assisting everybody to develop, as fully as 
possible, the knowledge and skills needed 1 to function, effectively *- n a rapidly changing 
society. Effective participation involves both the individual's need for fulfillment 
and the society's (system's) need for stability and cohesion. 

(2) At any point in the educational system, the individuals to be taught will 
differ in their capacities, interest, and backgrounds. To be effective, educational 
programs must be adjusted to take account of these differences. 
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(3) While providing for individual differences, the educator must recognize 
that the goals of education are both individual and social. 

(4) While in one sense the core of an educational program is instructional in 
content, in a broader sense, provision must be made for guidance and evaluation. 
Effective instruction (education) implies effective guidance (decision making) based 
on continuous evaluation (system feedback). 

(5) In a diverse and decentralized school system (such as the USA), details of 
organization and procedure will differ widely from school system to school system. 

It is the responsibility of analysts to provide for these differences and of 
administrators in each educational unit to determine details. 

(6) Individual schools and school systems must be provided with services that 
will increase the efficiency of their efforts to monitor their educational prbgrams—to 
the end that necessary decisions are informed and that educational programs aay be 
improved. 

(7) If provisions for evaluation are to be effective, they must be based on a 
sound conception of how individuals learn and how evidence of learning may be assembled 
and analyzed. Therefore, study of both the learning process and the evaluation process 

is a legitimate and required function of systems packages. 

It is within the framework suggested by these seven propositions that an outline 

for the development of a systems evaluation package, geared on the one hand to the 
needs of individuals* and on the other to the more global needs of the school- system, 
can be described. 

At the level of the individual, one promising system application would take 
advantage of the computer's capacity to monitor the performance of a great number 
of students and to select subsequent units of instruction on the basis of each 
student's response to questions embedded in previous instructional sequences and, at 
educational decision points^ retrieve up-to-date performance records for the teacher 
0 3r counselor. Such records may. in time assume some of the functions now served by 
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external examinations. Thus, the teaching and testing functions, which have never 

* 

been as intergrated as most theorists say they should, are likely to come even closer 
together in the future. It should be noted that we ar; s not referring here to CAI, but to 
what is called computer managed instruction. The functions described above might 
well be carried out by systems modules responsible for student record keeping and 
central predicition. 

At the level of the school system, educational administrators are likely to 
find themselves needing methods for collecting and summarizing data bearing on the 
over all functions of an educational system of ever-increasing complexity. In 
addition to information about the status of individual students at periodic inter- 
vals, those responsible for operating the schools need information about a variety 
of variables — characteristics of the staff, the nature of the tax base, a profile 
of community values, etc. — which should be taken into account in attempts to under- 
stand and evaluate the functions of the school system. System modules for resource 
allocation, scheduling, and the like might best carry out these aspects of the system 
evaluation. It. is clear that only through understanding evaluation requirements at 
this level and articulating them with the individual requirements specified above, can a 
system package be created that will satisify very real economic constraints as well as 
to sensitively adjust educational oppportunities to a wide variety of individuals. 

The two levels of consideration outlined above, then, suggest a range of 
evaluation concerns: from the evaluation of the individual to the evaluation of 

the educational systems which act upon the individual, or within which the individual 
interacts. The ability of the systems analyst to provide the necessary methods and 
the ability of the computer to facilitate cpmplex analyses of the many variables , offers 
promise tint the day is not far off when a usable systems evaluation may be operational. 



- 7 - 



Another very practical but important need is to avoid raising unrealistic 
expectations about what the new technology, particularly computers, can accomplish 
in a relatively short time period. There are no quick and easy solutions, and 
long-range commitment to systems applications is needed. Realistic expectations 
about what computers can do are vital, for example, analysts have not been as 
careful as they might to turn over to the computer only those tasks for which the 
machine is best suited and only those decision alternatives for which a completely 
defined and logical series of actions exists. Rather, they have all too often 
delegated to the computer system responsibility f° r directing and/or forcing decisions 
on the basis of faulty or incomplete information (Townsend, 1970). What is needed 
to resolve this dilemma is a totally intergrated system (perhaps designed on a modular 
basis) which can completely adapt to all current educational procedures and processes 
and which is designed so as not to preclude modification to satisfy future needs 
and demands. Development of this system will require time and extended effort. 

Before the system that we have described can be made operational, some rather 
important practical problems will have to be overcome. These problems are illustrated 
in the experience of one of the larger United States school systems in attempting 
to apply system concepts at all levels of its functioning. This program has been 
a long range effort which started three years ago, and, at this writingjthe program 
has not yet come to full fruition. Four critical problems which have hampered the 
development of a full systems approach: 

First, it has proved extremely difficult to get educators to state specific 
objectives in measurable terms. Sometimes, this problem has been made worse 
by the failure of the systems people to adequately communicate to educators 
what is needed. 

Second, it has been difficult to get educators to adhere to the system. 

There are really two problems here. The first is that educators work in a 
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political environment where decisions must be made to assuage this or that 

pressure group. The second problem is that decision making information 

has often not been timely. The last situation arises when hardware is 

installed without sufficient software and personnel backup. 

Third, present methods of relating input to output do not provide accurate 

forecasting. The complexities of the educational process, particularly in 

e large urban school system, are such that no simple model will suffice. 

* i 

Finally, even when adequate evaluation methods have been available, it has 
• been difficult to convince program managers that research and evaluation 
personnel should be involved in project development. 

What these reports suggest is that before any system can work in a real - 
setting, some rather serious thinking has to be given to the problem of educating 
both the designer and the user. To some extent, this is a problem that faces any 
systems designer. But, in an educational system which operates under considerable 
political constraint, at a level of complexity which is staggering, and in which 
many basic research questions are yet to be answered, the practical problems are 
not trivial. 

We started this paper by being critical of the systems analyst and the 
traditional educational researcher. This was deliberate for it has been clear 
for some time that neither were sufficiently aware, nor concerned, about the 
practical problems faced by the educator. It is our feeling that the failure 
of ''research * 1 and "systems 11 to cure the ills of education has been partially 
due as much to a failure of the physician to understand the problem as the patient's 
reluctance to take the cure. The amount of money, on both the Federal and Local 
levels, wasted on "systems" and research programs would astound and dismay most 
of us. The time is propitious for us, systems analysts and researchers alike, to 
seriously consider what it is we are about—and how pur efforts can contribute to 
our improvement in the quality life for all of us. 
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We close this monograph with two quotes; both of which come from persons 
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actively engaged in the business of systems design, at two quite different levels. 
The first, Robert Boguslaw is a systems analyst of some note. The second. 



Thomas Watson f , Jr. needs no identification. 



and. 



And perhaps the most notable difference to be. found between 
the classical system designers and their comtemporary couterparts 
(system engineers, data processing specialists, computer manufac- 
turers, and systems designers) consists precisely in the fact that 
the humanitarian bent has disappeared. The dominant value orien- 
tation of the utopian renaissance, eah best be described as 'effici- 
ency’ rather than 'humani tar ianism. 1 

(Bogimlaw, 1965, Page 202) 

In our recommitment to (humanitarian) values ... lies the key 

to technology's good use in our time — its service to the 

supreme end of education, the fulfillment, freedom and happiness of the 

individual human being. 

(Watson, 1968, Page 189) 
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Footnote 



Portions of this monograph are based on an invitational paper, "The educational 
system as a system," presented at the International Conference on Production 
Research, held as the University of Birmingham-, England, on April 5 through 
10, 1970, The ideas expressed here are those of the authors and do not 
necessarily represent positions of their respective organizations* 

Presentation of this paper was made possible by Grant GZ-1528 from the National 
Science Foundation to the first author* 
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